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ABSTRACT 

The ability of various Aerospace Materials 
Information Center (AMIC) information specialists to prepare search 
strategies for document retrieval was studied by providing ten 
typical search request statements to seven information specialists. 
Each specialist prepared search strategies independently. Significant 
variations occurred among the strategies, although even with these 
variations reasonably consistent document returns resulted. The 
experiment indicated that the proper interpretation of a search 
request and the conversion of the request to an appropriate search 
strategy even with a well-established thesaurus is a considerably 
more difficult task than indexing. In another experiment, the 
consistency of relevance judgments among information specialists was 
examined. Relevance judgment is more difficult than indexing and 
search strategy preparation. Agreement on nonrelevance is better than 
agreement on relevance. The system of relevance judgment, according 
to nonrelevant, partially relevant and relevant, appears optimum for 
AMIC. Communication between the requester, the information specialist 
and the retrieval system itself are of prime importance. Because of 
certain human interactions which must occur, some variability is 
unavoidable. However, current AMIC procedures of searching to achieve 
high recall and screening the results for relevance provide good 
results to the ultimate requester. (Author) 



XL 1890 lij 



AFML-TR-72-51 



U S DEPARTMENT OF HEALTH 
EDUCATION & WELFARE 
OFFICE OF EOUCATION 
THIS DOCUMENT HAS BEEN REPRO 

duced exactly as receiveo from 

THE PERSON OR ORGANIZATION ORIG 
iNATlNG IT POINTS OF VIEW OR OPlN 
IONS STATED DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU 
CATION POSITION OR POLICY 



DETERMINATION OF THE CONSISTENCY 
OF RELEVANCE JUDGMENTS AND THE 
RELIABILITY OF SEARCH STRATEGIES AMONG 
INFORMATION SPECIALISTS FOR THE AEROSPACE 
MATERIALS INFORMATION CENTER 



F. L. SCHEFFLER 
J. F. MARCH 

UNIVERSrrY OF DAYTON RESEARCH INSTITUTE 
DA YTON, OHIO 



TECHNICAL REPORT AFML-TR-72-51 



APRIL 1972 



Cv} Approved for public release; distribution is unlimited 

0 

1 

CO 

O AIR FORCE MATERIALS LABORATORY 

o air FORCE SYSTEMS COMMAND 

WR1GHT-PATTERSON AIR FORCE BASE, OHIO 




II 




FILMED FROM BEST AVAILABLE COPY 



NOTICE 



When Government drawings, specifications, or other data are used for 
any purpose other than in connection with a definitely related Government 
procurement operation, the United States Government thereby incurs no 
responsibility nor any obligation whatsoever ; and the fact that the govern- 
ment may have formulated furnished, or in any way supplied the said 
drawings specifications, or other data, is not to be regarded by implication 
or otherwise as in any manner licensing the holder or any other person or 
corporation, or conveying any rights or permission to manufacture, use, 
or sell any patented invention that may in any way be related thereto. 



% 



Copies of this report should not be returned unless return is required by 
security considerations, contractual obligations, or notice on a specific 
doc ument. 

AIR FORCE: 31-5.72/450 



O 

ERLC 



2 



FOREWORD 



This report was prepared by the University of Dayton Research 
Institute, Dayton, Ohio, under Air Force Contract F33615-71-C-1069. The 
work described herein was accomplished under Project 7381 "Materials 
Application" and Task 738103 "Materials Information Development, Collection 
and Processing. " The effort was administered under the direction of the 
Materials Information Branch, Materials Support Division, Air Force 
Materials Laboratory with Mr. Harold B. Thompson (AFML/ LAM) as 
Project Monitor. 

This is a final summary report and covers the work accomplished 
from 1 December 1970 through 30 November 1971. 

The authors acknowledge the efforts and contributions of Mrs. 

C. Marie Shanley, Mr. Howard H. Schumacher, Mr. Eugene R. Egan, 

Mr. David Z„ Winters as well as a number of students who participated in 
the program. 

This technical report has been reviewed and is approved. 




EDWARD DUGGER ft ft 

Chief, Materials Information Branch 

Materials Support Division 

Air Force Materials Laboratory 



TABLE OF CONTENTS 



j 

i 

| 




SECTION TITLE PAGE 

I. INTRODUCTION 1 

1. Description of AMIC System 1 

2. Summary of AMIC Procedures 2 

3. Indexer Consistency Analysis 2 

4. Purpose of Most Recent Experiments 5 

II. DOCUMENT RETRIEVALS AS A FUNCTION 7 

OF SEARCH STRATEGY 

1. Experimental Design 7 

2. Criteria for Analysis of Results 9 

3. Results 10 

III. CONSISTENCY OF RELEVANCE JUDGMENTS 29 

OF INFORMATION SPECIALISTS 

1. Experimental Design 29 

2. Criteria and Method for Analysis of Results 30 

3. Results 31 

IV. CONCLUSIONS 38 

V. DOCUMENT RETRIEVAL SYSTEM OPERATION 40 

1. -Input 40 

2. Searching 40 

3. Thesaurus Development 40 



v 

i 



4 



* 



i 



TABLE OF CONTENTS (Cont'd) 



REFERENCES 

APPENDICES 

APPENDIX I Definition of Task Numbers 

APPENDIX II Distribution of Personnel Time by 
Task Number 

APPENDIX III Subject Categories 

APPENDIX IV SDI Search Requests Processed 

1 December 1970 - 30 November 1971 

APPENDIX V Retrospective Search Requests Processed 
1 December 1970 - 30 November 1971 



PAGE 

43 



45 

47 

48 
54 

69 



APPENDIX VI Polymers 



90 



LIST OF TABLES 



i 

j 

i 



i 



\ 

{ 



ERIC 

hmffiiffBHaaaa 

V y, 



TABLE 

I 

II 

III -XII 

XIII 

XIV 

XV 

XVI-XEX 

XX 



PAGE 

Summary of Various Studies of Inter- and Intra- 4 

Indexer Consistency 

Retrospective Search Statements 8 

Retrospective Searches 1-10 14-23 

Combined Retrospective Searches 24 

Number of Individual Retrieval Terms Used in Search 25 

Commonality of Retrieval Terms Used In Searching 26 

Consistency of Relevance Judgments for Abstracts 33-36 
(Groups A-D) 

Differences of Relevance Judgments for Different 37 

Groups 



LIST OF FIGURES 



FIGURE 



PAGE 



1 Recall as a Function of Precision 27 

2 Recall as a Function of Individual Terms in 28 

Strategies 

3 Documents Indexed and Searches Processed by 42 

Year 



vii 

6 



i 



SECTION I 
INTRODUCTION 



1. DESCRIPTION OF AMIC SYSTEM 

The Information Systems Section of the University of Dayton Research 
Institute (UDRI) has established and presently maintains and operates a docu- 
ment retrieval. system in support of the Aerospace Materials Information 
Center (AMIC). The document retrieval system operated by the University 
of Dayton contains approximately 60, 000 documents concerning materials 
research and development with new accessions being made continually. The 
establishment, modification and operation of the document retrieval system 
are described in the following reports: RTD-TDR -63-4263 (AD 428 423) , 
AFML-TR -65-20 (AD 613 301) 2 , AFML-TR -66-36 (AD 633 614) 3 , AFML- 
TR -66-391 (AD 651 039) 4 , AFML-TR -67-379 (AD 666 462) 5 , AFML-TR - 
68-367 (AD 686 804) , AFML-TR-70-27 (AD 670 597) 7 and AFML-TR-71-1 1 
(AD 725 036)®. The present report describes the work performed from 
December 1970 to December 1971. 

The AMIC document retrieval system has been in operation with 
retrospective search capabilities since 1963. The purpose of the system 
is to provide scientific and technical information to qualified requesters 
in a timely and efficient manner. The information is supplied in the form 
of abstracts of documents pertinent to the search request; these abstract 
forms also contain complete bibliographic information including AMIC access 
number, DDC AD number or NASA N number, generating agency, report 
number, title, author, contract number (if applicable), and date of issue of 
the document. The documents themselves are available from the Materials 
Documentation Center (MDC) maintained at the Air Force Materials 
Laboratory (AFML). These documents are available on loan to the local 
requester, or may be referred to in the center. Abstracts of the documents 
are provided to the requesters. 

The AMIC document retrieval system is primarily concerned with the 
materials aspect of technical documents. Because of the concentration on 
materials, retrieval capabilities from a materials standpoint are very com- 
prehensive. Retrieval can be very specific, as, for example, all information 
on the alloy Aluminum 2024-T6, or retrieval could be as general as high 
temperature fatigue of all metals and alloys. Similarly, one may request 
information on boron reinforced Epon epoxy composites, or one may ask for 
aircraft structural applications of any composite material. 



Searches encompassing the entire range of materials information are 
regularly run by the UDRI in response to requests from technical personnel 
in the AFML. To ensure that the requester receives abstracts which are 
relevant to the request, all abstracts and index cards retrieved are screened 
for content by a UDRI information specialist to assess their relative pertinence 
to the originally-stated request. 

2. SUMMARY OF AMIC PROCEDURES 

The Aerospace Materials Information Center prepares documents by 
human indexing from the full text using a controlled vocabulary of index terms. 
The index terms serve as the storage and retrieval records for the computer 
system. The AMIC thesaurus is used to aid the indexer in the selection of 
appropriate terms. The indexers are familiar with the discipline represented 
by the document content, and they are carefully trained in indexing procedures. 
Automatic hierarchical posting of terms is provided to ensure that retrievals 
can be accomplished at the highest generic level desired. 

Searching is performed using Boolean logic linking of allowable index 
terms. A unique search cut-off feature provides a means of searching at 
various levels of specificity within a given search. This feature permits the 
search to be so constructed that the information specialist can be certain of 
achieving high recall. The cut-off feature also permits sorting of the search 
results by groups so that the most pertinent documents appear in the first 
groups. Retrospective search results are screened before they are submitted 
to the requester so that the requester receives only those abstracts judged 
relevant to his request, 

A Selective Dissemination of Information (SDI) program is in operation 
to provide abstracts on a recurring basis to the technical personnel in the 
AFML participating in the SDI program. The SDI results are run and dis- 
tributed for each update of the search file; each update represents the most 
recent input of indexed technical documents. 

The effectiveness of the AMIC system is dependent on three basic 
operations which represent interactions of the information specialists with 
the system. These are: indexing, search strategy formulation and the 
screening of retrospective search results. The consistency of indexing was 
investigated in a previous UDRI study. 5 

3. INDEXER CONSISTENCY ANALYSIS 

An evaluation procedure was developed to measure the indexing con- 
sistency of indexers in f’,..-- AMIC program. The evaluation was based upon 
the choice of index terms of an experienced information specialist, as a 
standard of indexing performance. The selection of index terms chosen by 
trained indexers was compared. 



Distinction was made between index terms which are essential for 
indexing a particular document and those which were helpful for retrieval 
but not essential to the information content. This distinction accommodates 
the indexing philosophy employed at UDRI governing the selection of index 
terms by which it is considered better to "overindex" (risking a false 
retrieval) than to "underindex" (risking nonretrieval of pertinent documents). 

Studies of intra- and inter indexer consistency were undertaken to 
determine the validity of using an experienced indexer's indexing as a 
reference standard and to assess the quality of indexing among various 
indexers. In 1966, 35 documents were selected which dealt primarily with 
chemistry as the primary subject. These documents were indexed inde- 
pendently by indexers and the information specialist. Several months later 
the experiment was repeated with another series of documents. Comparisons 
were made of essential terms, total terms, and time required for indexing. 

In 1967, the first set of 35 documents was recalled and indexing was performed 
by the two new indexer trainees and by two experienced indexers. Both 
experienced indexers were instructed to designate the essential terms for 
each indexed document. 

By comparing the indexing performed by the same indexer of the 
same document, a statistical correlation coefficient was obtained for essential 
terms which served as a measure of intraindexer reliability or consistency. 
The common essential terms were obtained between the two experienced 
indexers as a measure of interindexer consistency. The statistical correlation 
techniques used were the phi coefficient (J2f) and chi square (X^). 

The essential and total terms were compared by groups and the degree 
of correlation was determined between groups. For intraindexer consistency, 
the first group of terms was taken from one year's indexing and compared 
with the second group of terms from other year's indexing, all taken from the 
same set of documents. The results show statistically significant correlation 
for essential terms regarding intraindexer and interindexer consistency. 

Intraindexer consistency for the experienced information specialists 
was determined to be 78% overall and 92% for essential terms. When trained 
indexers were compared to experienced information specialists, the inter - 
indexer consistency was 62% overall and 83% for essential terms. 

The UDRI results compare reasonably well with the human indexing 
consistency studies reviewed by Mary Elizabeth Stevens in Automatic Indexing : 
A State-of-the-Art Report. Table I contains a list of results. This is 
especially true when the consistency values are derived from the statistics 
on essential terms. However, it is important to recognize the difficulties 
of comparison across differences in systems operation and subject areas. 
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SUMMARY OF VARIOUS STUDIES OF INTER- AND INTRA-INDEXER 

CONSISTENCY 9 





Inter - 


Intra- 




Indexer 


Indexer 




Consistency 


Consistency 


Macmillian and Welt 


18% 




Kyle 


70% 




AEC 


65%, 54% 


86% 


Jacoby and Slamecka 


20% 


50% 


Rodgers 


59% 




Painter 


62% 




Korotkin and Oliver 


53%, 54% 




UDRI 


62% (overall) 
83% (essential 
terms) 


78% (overall) 
92% (essential 
terms) 
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Several aspects of the UDRI AMIC operation enhanced the quality of indexing 
consistency. These include the extensive training program with which all 
indexers are introduced to the system, the well-developed AMIC thesaurus, 
and the comparatively specialized subject area of materials information. 

4. PURPOSE OF THE MOST RECENT EXPERIMENTS 

Students have been trained to index with reasonable consistency for 
the AMIC system. The most recent study sought to determine the ability of 
trained students to prepare search strategies independently with similar 
recall and relevance results and to examine their relevance judgments. 
Student performance was compared with professionals in a typical request 
and search situation. 

Various factors were recognized in studying relevance judgments. 

The amount of background in indexing and involvement with search requests 
could affect relevance judgments. Indexers from various disciplines were 
represented in the experiment. Since the same group of documents was 
used for eliciting relevance judgments, the formal academic training of the 
indexers might significantly affect their viewpoints in relevance judgments. 
Biology, chemistry, physics and engineering were disciplines represented 
among the indexers participating in the experiment. A fatigue factor is 
introduced by the quantity of abstracts for which relevance judgments must 
be made. Larger numbers of abstracts for a particular search for which 
judgments must be made may tend to reduce the discriminatory ability of 
persons applying relevance assessments. 

A review of the literature suggested that certain situations could be 
expected. Specifically, Saracevic * suggests that persons with a particular 
subject expertise and professional involvement tend to display greater dis- 
agreement in judging a set of documents than persons with a more general 
background. Furthermore, the persons with less subject knowledge of the 
documents tend to be more lenient in their judgments of relevance. Another 
influential factor in relevance judgments is the judges' understanding of the 
intended use of the relevant documents. 

In a more philosophical vein, Saracevic questions whether relevance 
judgment is a very subjective human process, or whether it has "associated 
with it regularity patterns that may be eventually utilized in the design of 
more effective IR systems. " Saracevic says "agreement as to what is not 
relevant may be expected to be greater than agreement as to what is relevant; 
judging relevance is not the same as judging non-relevance. " 



Lancaster claims "a relevant document is a document that is useful 
to the Requester in relation to the information need that prompted his request. 
Doyle warns that "there may be a great difference between relevance to a 
given request statement and relevance to a person's real information need." 
The ideal would be "to retrieve all and only those documents the searcher 
would regard as relevant to his need if he could personally inspect every 
document in the library. " In his exploratory study, Cuadra 13 considered 
the effect of academic and professional training on relevance judgments 
and concluded: "It appears likely that the disagreement at low experience 
levels stems from lack of knowledge, while that at higher levels reflects 
academic and interest specialization. " 

The AMIC experiments on the formulation of search strategies and 
on relevance judgments by various individuals were designed primarily to 
show the degree of consistency which could be expected by having different 
trained individuals perform these functions. It should be recognized that 
the experiments were specifically intended for the AMIC system which deals 
specifically with aerospace materials information. Furthermore, the 
individuals involved consisted of both trained students and prc'essionals . 
Although a number of factors were recognized as contributing to results 
which rnight be obtained, the primary purpose was to determine if adequate 
reliability in AMIC searching and screening operations could be obtained 
with the personnel assigned to these functions. 
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DOCUMENT RETRIEVALS AS A FUNCTION OF SEARCH STRATEGY 



The ability of information specialists to formulate effective search 
strategies in response to information requests is of great importance in the 
AMIC operations. As indicated earlier, the AMIC system depends on 
authorized keywords for indexing and retrieval. Retrieval is accomplished 
by Boolean logical linking of keywords with a cut-off feature, and access 
numbers corresponding to the various groupings of retrieval terms within 
the search are printed out. 

The students employed by AMIC receive training in the formulation 
of search strategies after they have mastered the indexing function. Experi- 
enced student personnel as well as professionals have been used to process 
technical search requests. In order to determine the consistency with which 
trained students and professionals can formulate searches, it was desired 
to investigate search strategies derived independently for a group of searches 
on various subjects. Differences in search strategies between individuals 
would be expected, but the degree to which various strategies are able to 
retrieve the same documents and, specifically, the ability to retrieve relevant 
documents (see Section III on consistency of relevance judgments) is important. 
Ideally, the requester should be provided with those abstracts from the entire 
file of documents which are relevant for his needs. In order to provide rele- 
vant documents to the requester, it is first necessary to retrieve approximately 
the same groups of documents regardless of the information specialist formu- 
lating the search strategy. The experiment on document retrievals as a 
function of search strategy was to determine the degree to which similar 
retrieval results would be attained by various information specialists. 

1. EXPERIMENTAL DESIGN 

Ten retrospective search requests were selected from our files of 
previous actual search requests. Care was exercised to include a variety 
of subject areas. A list of the requests used in this experiment appears 
in Table II. The requests were processed by six trained student indexers 
with varying backgrounds and one professional information specialist. 

The students' backgrounds were as follows: 



udent 


Field of Study 


Experience with AMIC 


SI 


Mathematics, Physics 


18 mo. 


S2 


Biology, Chemistry 


3 mo. 


S3 


Chemistry 


15 mo. 


S4 


Chemistry 


9 mo. 


S5 


Chemical Engineering 


3 mo. 


S6 


Chemistry 


3 mo. 
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TABLE II 

RETROSPECTIVE SEARCH STATEMENTS 



i 



i) 



Diffusion bonding, diffusion welding, 
of aluminum and aluminum alloys 



solid state bonding, or solid state welding 



2) The use of nematic liquid crystals in electro-optical display devices. There 
are three types of liquid crystals: nematic, cholesteric, and smectic. The 
first two are of special interest in that they are sometimes mixed for use in 
particular devices. The phenomenon of light-scattering by these materials 
is called dynamic scattering and the phase they are in is sometimes called 
mesomorphic. Temperature range -- 20°C to 80°C. 

3) The effect of reentry environments on the thermochemical erosion character- 
istics of ablative plastic composites. Rain/dust erosion effects to be excluded. 

4) Effect of heat treatment on microstructure and mechanical properties of titan- 



5) Laser cutting 

6) Flammability of organic and inorganic textile materials in all environments 
(air, oxygen, mixed gas system). 

7) Production, forming or joining metal matrix composites. Ni, A1 Ti Co 
or Nb reinforced by metal wires; B, SiC, A1 O , graphite etc. filaments;’ 
rods, thin films, or indigenous phase particles tin-situ reinforced com- 
posites). 



8) Gallium phosphide for ~ight emitting diodes, especially crystal growth and 
characterization. 



9) The use of photochromic and thermochromic compounds for nondestructive 
inspection systems. 

10) Adhesion of electroplated coatings to nonmetallic substrates (except ABS 
plastics or substrates). 
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Each indexer was instructed to formulate his own strategy independently for 
each request, following normal AMIC procedures. Index terms were selected 
using the AMIC Thesaurus as a guide. Boolean AND. OR, NOT logic was 
applied to the grouping of terms used for retrieval in the search strategy. 

The students also utilized the cutoff feature, which enables the returns to be 
sorted into groups in order of anticipated relevance and allows the separation 
of different sub-topics. All of the returns were screened for relevance by an 
information specialist who was not involved in the strategy formulation aspect 
of the experiment. 

Certain assumptions were made in the conduct of the experiment. 

First, it was assumed that among all the strategies prepared all of the relevant 
documents would be represented. This assumption is of course an approximation, 
since practical considerations obviate the possibility of manually screening the 
entire file of 60, 000 documents. A second assumption was that the relevance 
judgments applied to the search returns by an independent information specialist 
were sufficiently consistent to serve as a basis for recall and precision. Third, 
because of the cutoff feature, certain search strategy results included groups 
representing large numbers of documents known to have low relevance. This 
situation is consistent with the AMIC philosophy of effecting high recall, but in 
those cases where the screening process would have been prohibitive and it was 
known that very low relevance would have resulted, these groups were omitted 
from screening. 

2. CRITERIA FOR ANALYSIS 

The primary consideration for analyzing the results was the ability of 
retrieval strategies prepared by different individuals to elicit from the entire 
document file similar document subsets. In particular, it would be most 
desirable for a high degree of consistency to occur among the various individuals 
for those documents actually considered relevant. 

Another cons ideration for assessing the performance of various search 
strategies is the precision and recall performance. Precision is defined as 
the proportion of relevant documents retrieved to the total documents retrieved 
and recall is defined as the proportion of relevant documents retrieved to the 
total available relevant documents; the approximation of total available rele- 
vant documents should be kept in mind. 

A third criterion is the degree to which the same retrieval terms 
were used among the various individuals for retrieval in response to a given 
request. A related factor is the number of distinct individual retrieval terms 
used by each person. It should be noted that with the AMIC search program 
a given term can be used more than once to achieve sorting of the documents 
retrieved. 



3. 



RESULTS 



The number of documents retrieved by each person's strategy for 
all ten searches is given in Tables III - XII. These tables also indicate the 
precision and recall results. The commonality of the relevant retrievals 
is shown by a frequency distribution. The commonality refers to the number 
of searchers out of seven retrieving the same relevant documents. Table 
XIII gives the combined results for all ten searches. Precision, also called 
relevance, is defined as the ratio of the relevant documents retrieved to the 
total number of documents retrieved. Recall is defined as the ratio of the 
relevant documents actually retrieved to the number of relevant documents 
available; in this study the number of relevant documents available is assumed 
to be the summation of the relevant documents retrieved by all searchers. 

The search results indicate a considerable variation between individual searche 
For certain searches there is surprising uniformity, whereas for others much 
variation occurs both with search strategy and with results obtained. In 
analyzing the search strategies it was found that in some cases the inclusion or 
exclusion of a particular term can make a great difference in retrieval results. 
This was particularly true in the case of RS 9, in which Searcher S5 included 
the term LIQUID CRYSTALS. This term retrieved many relevant documents 
which were not retrieved by the others. In Search RS 1, one individual used 
more general terminology for retrieval resulting in a large number of retrieved 
documents. 

Referring to Table XIII, one can see that there is relatively little 
difference between precision and recall ratios among various individuals 
across the ten searches. As would be expected, 9 the recall tends to vary 
inversely with the precision. This trend is shown in Figure 1. 

To evaluate the effect of the number of individual terms in the search 
strategy on precision and recall, terms used by each searcher for each 
request was examined as shown in Table XIV. From each searcher's 
average number of individual terms in relation to average recall as shown 
in Figure 2, recall increases with more terms in the strategies, but this 
effect becomes less pronounced as the overall number of individual terms 
increases. The recall achieved by searcher S3 using 10.2 terms per search 
strategy does not significantly exceed that of another searcher P.l using 8. 0 
terms per search strategy. It should also be noted that the number of 
retrieval terms is only an indication of the retrievals to be anticipated. 

The posting density of documents under the terms also has an effect. 

To obtain an indication of the degree of consistency among searchers, 
a frequency distribution of search terms was determined as shown in Table XV. 
According to this table, the frequency with which nearly all searchers used 
the same term for retrieval was fairly high. On the other hand, for many 
searches a number of retrieval terms was used by only one or two of the 
searchers. It is interesting to note that even with a fairly high degree of 
variability in the terms used for retrieval, the precision and recall results 

10 



for most individuals was nearly the same. This situation can be attributed 
in large measure to the fact that many of the same documents are retrieved 
even with different strategies. This, in turn, reflects the AMIC philosophy 
of indexing documents in some depth (about 20 terms per document before 
automatic hierarchical posting), to index with specificity and to use as many 
portions of the document as necessary (up to the complete text of the document) 
as a source of index terms. 

Since search strategies and the corresponding results varied among 
individuals, it was considered whether experience might influence average 
precision and recall. However, the two most experienced student searchers 
(S3 and SI) tended respectively toward the highest and lowest recall rates. 

The same situation is true for two of the less experience searchers. S2 
had the highest and S6 the lowest recall of the entire group. The professional 
information specialist PI was among the searchers achieving high recall. 

The consistency of results of different searchers on the same request 
varied considerably for different requests, indicating that perhaps the nature 
and wording of the statement are important. Some requests are concise and 
fairly straightforward, and the principal terms in the request used are also 
AMIC thesaurus terms. Other requests, however, leave considerable latitude 
for variation in term selection. This is due in some cases to ambiguous 
statement of the request. 

Search RS 4 is one of the more direct and clear ly-worded requests. 

All of its key terms (HEAT TREATMENT, MECHANICAL PROPERTIES, 
MICROSTRUCTURE, and TITANIUM -Al-V) are AMIC terms, so that little 
imaginative effort is necessary for expression of these concepts. As a 
result, for an average of 5. 3 individual terms used for this search, these 
4 AMIC terms were used by at least 6 searchers, (3 of these terms were 
used by all 7). Four searchers had identical returns as a result of using 
a variation of this strategy: 



The other three searchers used a few other related terms in addition to those 
in the above strategy. One such strategy was: 



AND 

AND 

AND 

OR 



TITANIUM -Al-V 
HEAT TREATMENT 
MECHANICAL PROPERTIES 
MICROSTRUCTURE 



AND 

AND 

OR 

OR 

OR 

AND 

OR 



TITANIUM -AL-V 
HEAT TREATMENT 
CHARRING 
QUENCHING 
HEATING 

MICR OSTR UC TUR E 
MECHANICAL PROPERTIES 
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As a result of this consistency in strategy formulation, 39 ( 60%) of the 65 
relevant documents returned were found by at least 6 searchers. Twelve 
of these documents were retrieved by everyone. This consistency is 
reflected in the recall statistics. Five of seven searchers had recall 
higher than 0.70, and all seven achieved recall greater than 0. 50. 

When the results from Search RS 3 is examined, a different trend is 
observed. Since the phrase "thermochemical erosion" is not directly repre- 
sented in the AMIC Thesaurus, many different terms (EROSION, SPALLING, 
ABRASION, DECOMPOSITION, SUBLIMATION, VAPORIZATION, CHEMICAL 
REACTIONS, etc. ) were used by various individuals to express this concept. 
The average strategy used 9. 7 individual terms for this request; only three 
of these terms were common to six or more searchers. Some strategies 
were short and restrictive, like the following: 



Other strategies, however, were quite long and allowed for much broader 
coverage of the subject: 



NOT 

NOT 

NOT 

AND 

OR 

AND 

AND 

AND 



RAIN 

SAND 

WEATHERING 

RE-ENTRY 

RE-ENTRY VEHICLES 
ABLATION 

POLYMER COMPOSITES 
EROSION 



NOT RAIN 

NOT SUSPENSIONS 

AND POLYMER COMPOSITES 

AND RE-ENTRY 

OR RE-ENTRY VEHICLES 

AND ABLATION 

OR DECOMPOSITION 

OR CHEMICAL REACTIONS 

OR VOLATILIZATION 

OR SUBLIMATION 

OR VAPORIZATION 

AND EROSION 

OR SPALLING 

OR SURFACES 

OR RE-ENTRY 

AND EROSION 

OR SPALLING 

OR SURFACES 

AND RE-ENTRY 

AND EROSION 



12 



18 



* 



Because the strategy formulations differed so widely for this request, there was 
little consistency in results. Of a total of 35 relevant documents retrieved, 
only 2 (6%) were common to more than two different searchers. 20 relevant 
documents were retrieved only once. As a result of this inconsistency, recall 
figures were low: 4 searchers had a recall of less than 0.25 while only one 
searcher attained a recall of more than 0. 5. 

The variation of consistency in search strategies and results with the 
nature of the search request seems to indicate that, in some cases, clarification 
of the request statement would have improved search results. Normally, the 
searcher is able to contact the original requester if the request statement is 
not clear. However, due to the design of this experiment, the searchers were 
required to work only from the written request. 
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TABLE III 

Search rs* i 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
R etr ieved 


Precision 


Recall 


SI** 


52 


20 


.38 


.32 


S2 


243 


60 


.25 


. 97 


S3 


145 


30 


. 21 


.48 


S4 


69 


•20 


.29 


.32 


S5 


32 


13 


.41 


.21 


S 6 


1 


1 


1 . 00 


. 02 


*** 

PI 


79 


21 


.27 


.34 



I 

t 




ERLC 

•.'I’LTmITLJ 



Commonality of Retrieval Among 
Searchers 


No. of R elevant Documents 
Retrieved 


7 out of 7 


1 


( 2 %) 


6 out of 7 


8 


(13%) 


5 out of 7 


8 


(13%) 


4 out of 7 


2 


( 3%) 


3 out of 7 


1 


( 2 %) 


2 out of 7 


17 


(27%) 


1 out of 7 


25 


(40%) 


& & & & & 

Total relevant documents 


62 





Retrospective Search The total relevant documents refers 

to the approximation that the summation 
Student of relevant documents retrieved by all 

seven searchers represents the total 
Professional available relevant documents. 

Number of searchers out of seven 
who retrieved the same document 14 




RS = 
** 

S = 
P = 






; 

j 

i 

1 

i 

! 

j 






TABLE IV 
SEARCH RS 2 



Individual 

Searcher 


Total Documents 
R etrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


18 


10 


.55 


.91 


S2 


NA 


NA 


NA 


NA 


S3 


20 


6 


. 30 


,55 


S4 


18 


10 


.55 


.91 


S5 


9 


6 


.66 


.55 


S6 


NA 


NA 


NA 


NA 


PI 


18 


10 


.55 


.91 



NA = Not available 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


5 out of 5 


5 


(45%) 


4 out of 5 


1 


( 9%) 


3 out of 5 


4 


(36%) 


2 out of 5 


0 


( 0%) 


1 out of 5 


1 


( 9%) 


Total relevant documents 


11 






15 







TABLE V 
SEARCH RS 3 



Individual 

Searcher 


Total Documents 
R etrieved 


Relevant Documents 
R etrieved 


Precision 


Recall 


SI 


3 


2 


. 67 


.06 


S2 


53 


20 


. 37 


.57 


S3 


13 


6 


.46 


. 17 


S4 


15 


11 


. 73 


.31 


S5 


7 


3 


.43 


.09 


S6 


2 


2 


1. 00 


.06 


PI 


20 


11 


.55 


.31 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


0 


( 0%) 


6 out of 7 


1 


( 3%) 


5 out of 7 


1 


( 3%) 


4 out of 7 


0 


( 0%) 


3 out of 7 


0 


( 0%) 


2 out of 7 


13 


(37%) 


1 out of 7 


20 


(57%) 


Total relevant documents 


35 
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TABLE VI 
SEARCH RS 4 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


123 


48 


.39 


. 74 


S2 


123 


48 


.39 


.74 


S3 


205 


43 


.21 


.66 


S4 


134 


51 


.38 


.78 


S5 


123 


48 


.39 


. 74 


S6 


128 


35 


.27 


. 54 


PI 


123 


48 


.39 


.74 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


12 


(18%) 


6 out of 7 


27 


(41%) 


5 out of 7 


9 


(13%) 


4 out of 7 


0 


( 0%) 


3 out of 7 


2 


( 3%) 


2 out of 7 


9 


(13%) 


1 out of 7 


6 


( 9%) 


Total relevant documents 


65 






.17 




TABLE VII 



SEARCH RS 5 



I 

1 

i 

i 

( 

i 

i 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


4 


3 


.75 


.33 


S2 


19 


9 


.47 


1. 00 


S3 


19 


9 


.47 


1. 00 


S4 


19 


9 


.47 


1. 00 


S5 


19 


9 


.47 


1. 00 


S 6 


19 


9 


.47 


1. 00 


PI 


16 


8 


. 50 


.89 





Commonality of Retrieval Among No. of Relevant Documents 



Searchers 


Retrieved 


7 out of 7 


3 


(33%) 


6 out of 7 


6 


(67%) 


5 out of 7 


0 


( 0 %) 


4 out of 7 


0 


( 0 %) 


3 out of 7 


0 


( 0 %) 


2 out of 7 


0 


( 0 %) 


1 out of 7 


0 


( 0 %) 



Total relevant documents 9 




18 
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TABLE VIII 
SEARCH RS 6 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents Precision 

R etrieved 


Recall 


SI 


26 


14 


. 54 


. 67 


S2 


26 


14 


. 54 


. 67 


S3 


66 


19 


.29 


. 90 


S4 


20 


10 


. 50 


.48 


S5 


7 


4 


. 57 


. 19 


S6 


26 


14 


. 54 


. 67 


PI 


21 


11 


. 52 


.52 



I 



I 



$ 



i 







Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


3 


(14%) 


6 out of 7 


4 


(19%) 


5 out of 7 


4 


(19%) 


4 out of 7 


3 


(14%) 


3 out of 7 


2 


(10%) 


2 out of 7 


5 


(24%) 


1 out of 7 


0 


( 0%) 


Total relevant documents 


21 





19 




* 



TABLE DC 
SEARCH RS 7 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


51 


25 


.49 


.45 


S2 


15 


7 


. 47 


. 13 


S3 


82 


37 


. 45 


.66 


S4 


102 


37 


. 36 


.66 


S5 


75 


34 


.45 


. 60 


S6 


40 


17 


.42 


.30 


PI 


109 


48 


. 44 


00 

ON 



ERJC 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


1 


( 2%) 


6 out of 7 


11 


(20%) 


5 out of 7 


11 


(20%) 


4 out of 7 


8 


(14%) 


3 out of 7 


4 


( 6%) 


2 out of 7 


13 


(23%) 


1 out of 7 


8 


(14%) 


Total relevant documents 


56 





20 



S?6 



■ 



TABLE X 
SEARCH RS 8 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


6 


5 


. 83 


.20 


S2 


48 


25 


. 52 


1.00 


S3 


33 


18 


. 54 


.72 


S4 


11 


7 


. 64 


.28 


S5 


11 


7 


. 64 


.28 


S6 


11 


7 


. 64 


.28 


PI 


48 


25 


. 52 


1.00 



f 

i 

i 



f 




ERIC 

MlltggiailiiU 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


5 


(20%) 


6 out of 7 


2 


( 8%) 


5 out of 7 


0 


( 0%) 


4 out of 7 


0 


( 0%) 


3 out of 7 


11 


(44%) 


2 out of 7 


7 


(28%) 


1 out of 7 


0 


( 0%) 


Total relevant documents 


25 





i 

! 



21 




* 



■ 



TABLE XI 
SEARCH RS 9 



Ind ividual 
Searcher 


Total Documents 
Retrieved 


Relevant Documents Precision 
Retrieved 


Recall 


SI 


2 


1 


. 50 


. 05 


S2 


2 


1 


. 50 


. 05 


S3 


4 


3 


. 75 


. 14 


S4 


4 


3 


. 75 


. 14 


S5 


40 


19 


. 47 


.91 


S 6 


2 


1 


. 50 


. 05 


PI 


2 


1 


. 50 


.05 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


0 


( 


0%) 


6 out of 7 


1 


( 


5%) 


5 out of 7 


0 


( 


0%) 


4 out of 7 


0 


( 


0%) 


3 out of 7 


0 


( 


0%) 


2 out of 7 


1 


( 


5%) 


1 out of 7 


19 


(90%) 


Total relevant documents 


21 
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TABLE XII 
SEARCH RS 10 



Individual 

Searcher 


Total Documents 
Retrieved 


Relevant Documents 
Retrieved 


Precision 


Recall 


SI 


32 


5 


.16 


.26 


S2 


32 


5 


.16 


.26 


S3 


76 


15 


.20 


.79 


S4 


44 


9 


.20 


.47 


S5 


8 


1 


.13 


. 05 


S6 


10 


2 


.20 


. 11 


PI 


36 


5 


. 14 


.26 





Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


1 


( 5%) 


6 out of 7 


1 


( 5%) 


5 out of 7 


1 


( 5%) 


4 out of 7 


2 


(11%) 


3 out of 7 


0 


( 0%) 


2 out of 7 


2 


(11%) 


1 out of 7 


12 


(63%) 



Total relevant documents 19 



TABLE Xm 

COMBINED RS SEARCHES 



Individual 

Searches 


Total 

Documents 

Retrieved 


Relevant 

Documents 


Precision 
(document 
basis) * 


Recall 

(document 

basis) 


Precision 

(search 

basis)** 


Recall 

(search 

basis) 


SI 


317 


133 


.42 


.41 


.53 


.41 


S2 


561 


189 


. 34 


. 58 


.41 


.61 


S3 


663 


186 


. 28 


. 57 


. 36 


.60 


S4 


436 


167 


,38 


. 52 


.49 


. 55 


S5 


331 


144 


.43 


.44 


.46 


.46 


S6 


239 


88 


. 37 


.27 


. 56 


.34 


PI 


472 


188 


.40 


. 58 


.44 


.59 



! 



9 

ERIC 

•i'l’LTlmlTl J 



Commonality of Retrieval Among 
Searchers 


No. of Relevant Documents 
Retrieved 


7 out of 7 


26 


( 8%) 


6 out of 7 


66 


(20%) 


5 out of 7 


35 


(11%) 


4 out of 7 


19 


( 6%) 


3 out of 7 


20 


( 6%) 


2 out of 7 


68 


(21%) 


1 out of 7 


90 


(28%) 


Total relevant documents 


324 





ij; 

Document basis refers to the precision and recall obtained from the total number 
of documents retrieved for ten searches 

Search basis refers to the mean average precision and recall for ten searches 
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Results based on five searchers* results for this search 



AVERAGE RECALL FOR INDIVIDUALS 



0.7 





0.3 0.4 0.5 0.6 

AVERAGE PRECISION FOR INDIVIDUALS (SEARCHES RSI-RS 10) 

* 

Figure 1. Recall as a Function of Precision 
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SECTION III 



CONSISTENCY OF RELEVANCE JUDGMENTS 
OF INFORMATION SPECIALISTS 



The consistency of relevance judgments of various individuals 
associated with the screening of AMIC searches is an important factor, 
because the material received by the requester is dependent on the rele- 
vance judgments assigned by the information specialist. The AMIC system 
philosophy of searching is to strive for high recall to ensure that failure 
to retrieve relevant documents does not occur. With high recall, there is 
also a tendency toward low relevance. To provide the requester with useful 
documents, all search results are screened, and a judgment of relevance is 
applied by the information specialist. The judgments are indicated as pro- 
bably relevant, possibly relevant and not relevant. The abstracts and index 
cards corresponding to the retrieved documents serve as the basis for 
judgment. Only those abstracts deemed either probably relevant or possibly 
relevant are actually forwarded to the requester, thus obviating the need for 
him to look through nonrelevant material. A prime consideration in this 
process is the consistency with which relevance judgments are applied by 
various information specialists. 

1. EXPERIMENTAL DESIGN 

The basic philosophy of the experimental design was to simulate 
actual AMIC operations as closely as possible. Four search results were 
selected. These were composed of four groups of retrieved access numbers 
designated as Group A, Group B, Group C and Group D. For each group, 
a statement of the search request was provided. The retrieved abstracts 
were compared one at a time with the search request statement and a 
relevance judgment of probably relevant, possibly relevant, and nonrelevant 
was indicated for each abstract by each individual participant in the experiment. 
Six trained students in technical disciplines took part in the experiment. 

Group C was also screened by two professional information specialists. The 
search request statements were as follows: 

Group A Interest is in vibration damping, especially of aircraft, 

aerospace vehicles and components thereof. Keywords: 

205 documents vibration damping, vibration theory, infrasonic vibration, 
mechanical vibration, flexural vibration, vibration damp- 
ing, vibration measurement, vibration analysis, vibration 
synthesis vibration simulation, vibration stress, acoustic 
damping, acoustic shielding, noise, structural vibration. 



35 



29 



Group B The entire area of hypervelocity impact is of interest 

including materials, testing methods and theoretical 
196 documents studies. Hypervelocity is being explored in connection 
with research on the meteoroid hazard to spacecraft, 
impact craters on planetary surfaces, material pro- 
perties under conditions of shock loading. Meteoroid 
protection, debris clouds, impact craters, shockwaves, 
shock hydrodynamics, shock physics, gas guns, powder 
guns, sensors for impact measurement, computer 
programs and simulation of impact, rain erosion, high 
pressure and velocity behavior of materials, explosions 
and effects on materials. Nuclear physics is not of 
interest. “ 



Group C Interest is in protective coatings to prevent corrosion. 

Electrochemical and stress corrosion are of particular 
213 documents concern. Inorganic polymers, semi-organic polymers, 
silicones, phosphonitrile materials as nonrigid coatings 
are of interest. High temperature environments are of 
interest, it is desired to develop a semiflexible high 
temperature resistant coating. The area of interest is 
research and development oriented. 



Group D 
112 documents 



Interest is in erosion resistant coatings both metallic and 
nonmetallic; the application is for turbine blade coatings 
in gas turbine engines. 



CRITERIA AND METHOD FOR ANALYSIS RESULTS 



The basic criterion for analyzing the results was the degree of agree- 
ment among the participants in the experiment. The three-value relevance 
judgment conditions represented by probably relevant ( v^or C), possibly 
relevant (X), and nonrelevant (O) were used. For any two given individuals, 
the following combinations of values could occur, namely, C-C, X-X, O-O, 
C-O, X-O, X-C. To analyze the consistency of the relevance judgment 
results of the six students, the frequencies of the six possible combinations 
were derived by considering two students at a time until all possible com- 
binations of students had been exhausted. In this way, a composite frequency 
distribution for all students was calculated and presented in terms of per- 
centages. This procedure was repeated for all four searches. In the case 
of the two professional indexers, only a single calculation was needed, since 
only two individuals and one search group was involved. 

Another aspect of interest was the degree of variation between the 
various groups of documents screened. This degree of variation was 
expressed in terms of the standard deviation among the four searches for 
each relevance judgment combination. The standard deviation represents 
the tendency of data to cluster about the mean value. Eighty-four percent 
of the data will fall within two standard deviations of the mean value. There- 
fore the smaller the standard deviation, the less variability of data occurs. 

30 
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3. 



RESULTS 



The results are presented in Tables XVI-XX. These tables show the 
frequencies with which indexers taken two at a time agree and disagree on 
their relevance judgments. The columns are ordered from left to right such 
that the left hand column represents the best situation (agreement that the 
document is nonrelevant) and the right hand column represents the worst 
situation (one person judges that the document is probably relevant, the other 
judges that the document is nonrelevant). 

For Group C shown in Table XVIII two professionals also screened the 
document group for relevance. A comparison of the performance of pro- 
fessionals for Group C with the performance of the composite group of students 
tends to indicate that the professionals tended to agree better on judgments 
of nonrelevance. However, it is surprising to see that the frequency of 
disagreement as indicated by the frequency of O-C is actually higher than 
for the student composite. 

Table XX shows that there may be some difference between searches 
as indicated by the standard deviations for the six possible combinations of 
agreement on relevance. The standard deviation for the 0-0 condition is 
very small. This finding suggests that differences in subject matter and 
search statements probably have little effect on the composite relevance 
judgments . 

The normal procedure in the AMIC operation is for the information 
specialist performing the search to screen the search results for relevance 
and to forward probably relevant and possibly relevant abstracts to the 
requester. The differentiation is made primarily as an aid to the requester 
so that he can decide if he wishes to review all the abstracts or just those 
indicated as having close relevance to his request. Thus, both C and X 
abstracts are forwarded. Nonrelevant abstracts are not sent in order to 
preclude the need for the requester to have to sort through them. Therefore, 
from the standpoint of reviewing the results of the experiment, the situation 
in which disagreement of relevance value occurs between a C and an X is not 
as significant as disagreements resulting in O-C or O-X, since the latter 
situation would cause the document either to be sent or not to be sent. 

The experiment shows that there is fairly good agreement on non- 
relevant abstracts but there is significantly less agreement regarding the 
attributes which cause an abstract to be judged probably relevant or possibly 
relevant. An additional factor which is not accounted for in this experiment 
is the judgment of relevance which would have been assigned by the requester 
himself. However, previous UDRI studies 7 ’ indicated that generally 
requesters were satisfied with the relevance of material received. The 
findings of this experiment show that although the judgment of nonrelevance 



IS fairly good, the judgment of relevance is much more difficult. The 
practical result of the findings is that the group of abstracts actually for- 
warded to the requester for any given search request will vary depending 
on the information specialist handling the request. 

An informal study was made of the possibility of using a two-value 
relevance judgment system for the screening of abstracts; that is, the 
abstract must be judged either as relevant or nonrelevant. The results 
of this brief study indicated that judging with a two-value situation required 
more time in decision making and usually resulted in fewer abstracts which 
would have been sent to the requester. 

With reference to other studies on relevance, *5, 16, 17 

our results tend to confirm findings of some others that relevance judgments 
are somewhat elusive. Many factors can contribute to variations in relevance 
judgments including the experience and background of the individual judge, the 
understanding and interpretation of the request statement, the content of the 
abstracts being screened, and others. There seems to be a difference in 
judging nonrelevance in contrast to judging relevance, as indicated by 
Saracevic . 



TABLE XVI 



I 
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CONSISTENCY OF RELEVANCE JUDGMENTS FOR ABSTRACTS 

IN GROUP A (205 ABSTRACTS) 



Pairing 


Relevance Judgment 


Condition 


(Frequency %) 






0-0 


C-C 


X-X 


X-C 


O-X 


O-C 


* SI 


- S2 


64. 4% 


5.4% 


7.9% 


8.3% 


15. 1% 


a. 9% 


SI 


- S3 


38. 1% 


8.8% 


4. 9% 


11.2% 


25. 8% 


11.2% 


SI 


- S4 


69. 9% 


5.4% 


2.4% 


7.8% 


9.8% 


4. 9% 


SI 


- S5 


63. 5% 


7.8% 


1.0% 


10.7% 


10. 2% 


6. 8% 


si 


- S6 


37. 1% 


8.8% 


2.9% 


11.2% 


17. 5% 


22.4% 


52 


- S3 


35. 1% 


9.8% 


3.4% 


11.2% 


29. 8% 


11.7% 


S2 


- S4 


65. 4% 


7.8% 


2.0% 


3.9% 


14. 6% 


6.3% 


S2 


- S5 


60. 0% 


9.3% 


3.4% 


9.7% 


10.2% 


11.2% 


S2 


- S6 


34. 6% 


11.2% 


3. 9% 


7.8% 


19.0% 


23.4% 


S3 


- S4 


36. 1% 


11.2% 


2.4% 


5.4% 


29. 8% 


15. 1% 


S3 


- S5 


35. 1% 


17. 5% 


3.4% 


7.3% 


26. 3% 


10. 2% 


S3 


- S6 


27. 3% 


23.4% 


8. 3% 


ra. 5% 


17. 1% 


5.4% 


S4 


- S5 


63. 0% 


8.8% 


1.0% 


5.4% 


8. 8% 


13. 2% 


S4 


- S6 


38. 0% 


11.7% 


2.0% 


6.3% 


17.5% 


24. 9% 


S5 


- S6 


34. 2% 


16.6% 


0.5% 


9.3% 


18. 0% 


21.4% 


Mean percent 


46. 8% 


10.9% 


3.0% 


8.7% 


17. 9% 


12. 8% 



* S = Student 
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TABLE XVII 



CONSISTENCY OF RELEVANCE JUDGMENTS FOR ABSTRACTS 

IN GROUP B (196 ABSTRACTS) 



Pairing 


Relevance Judgment Condition 
0-0 C-C X-X X-C 


(Frequency 

O-X 


%) 

O-C 


* SI 




S2 


45. 4% 


18. 9% 


3.6% 


7. 1% 


17. 9% 


7. 1% 


SI 




S3 


33. 7% 


20. 9% 


9. 7% 


9.2% 


21.9% 


4.6% 


SI 




S4 


48. 0% 


13. 8% 


1.0% 


8. 7% 


20. 4% 


8. 2% 


si 


“ 


S5 


48. 5% 


14. 8% 


2.0% 


4. 1% 


18.4% 


12. 2% 


SI 


- 


S6 


44. 9% 


19.4% 


5. 1% 


8.2% 


15.8% 


6. 6% 


S2 


- 


S3 


37. 8% 


18. 9% 


4.6% 


6.6% 


9.7% 


10. 2% 


S2 


- 


S4 


59. 7% 


13. 8% 


1.0% 


6. 6% 


9.7% 


9.2% 


S2 


- 


S5 


59.2% 


14. 8% 


0. 0% 


3. 1% 


10.7% 


12.2% 


S2 


- 


S6 


54. 1% 


19.9% 


3.. 1% 


5. 1% 


10.2% 


7.7% 


S3 


- 


S4 


38. 8% 


14. 8% 


1.0% 


6 . 6% 


28. 1% 


10. 7% 


S3 


- 


S5 


39. 8% 


16.3% 


2.6% 


1.5% 


25. 5% 


14. 3% 


S3 


- 


S 6 


36.7% 


19.4% 


4. 6% 


8. 7% 


21.9% 


8. 7% 


Si 


- 


S5 


67. 3% 


13.3% 


0 .. 5% 


3. 6% 


8.2% 


7. 1% 


S4 


- 


S 6 


57. 7% 


13. 8% 


1.5% 


7.7% 


9.7% 


9. 7% 


S5 


- 


S6 


58. 2% 


15.8% 


0. 5% 


4. 1% 


10.. 7% 


10. 7% 



Mean percent 


48. 6% 


1 6. 6% 


2.7% 


6. 1% 


16.7% 


9.3% 



* S = Student 



40 



34 



table xviii 



CONSISTENCY OF RELEVANCE JUDGMENTS FOR ABSTRACTS 

IN GROUP C (213 ABSTRACTS) 



Pairing 


Relevance Judgment 


Condition 


(Frequency 


%) 






0-0 


C-C 


X-X 


X-C 


O-X 


O-C 


♦ SI 


- S2 


53. 1% 


3. 8% 


3.3% 


4.2% 


27. 7% 


8.0% 


SI 


- S3 


54. 9% 


5.2% 


a. 3% 


3.3% 


22. 5% 


10. 8% 


SI 


- S4 


53. 1% 


10. 3% 


4. 7% 


7.0% 


17.4% 


7.5% 


SI 


- S5 


51.2% 


9. 9% 


5. 6% 


6.6% 


18. 3% 


8.5% 


si 


- S6 


30. 0% 


9. 9% 


7.0% 


12.2% 


24. 4% 


16. 4% 


S2 


- S3 


64. 8% 


2.8% 


4. 7% 


3.8% 


18.3% 


5.6% 


S2 


- S4 


56. 8% 


4.7% 


4. 2% 


6. 6% 


17. 8% 


9. 9% 


S2 


- S5 


56. 3% 


3. 8% 


4.2% 


8.9% 


17. 8% 


8. 9% 


S2 


- S6 


32. 9% 


4. 2% 


4. 2% 


13. 1% 


28. 2% 


17. 4% 


S3 


- S4 


55. 9% 


5.2% 


1.4% 


6.6% 


17.4% 


13. 6% 


S3 


- S5 


54. 0% 


, 5.2% 


1.4% 


6.6% 


19.7% 


13. 1% 


S3 


- S6 


33. 8% 


7.0% 


2.8% 


11.3% 


26., 8% 


18.3% 


S4 


- S5 


60. 6% 


17. 4% 


12.2% 


2.3% 


3.8% 


3. 8% 


S4 


- S6 


29. 1% 


13. 1% 


5.2% 


10.3% 


24. 4% 


17. 8% 


S5 


- S6 


29. 6% 


13. 6% 


6.1% 


11.3% 


23. 9% 


15. 5% 


+ PI 


- P2 


69. 0% 


2.3% 


1.4% 


3.8% 


8. 9% 


14. 6% 


Mean percent 


47. 7% 


7. 7% 


4.7% 


7. 6% 


20. 6% 


11.7% 


Professional 


69. 0% 


2.3% 


1.4% 


3.8% 


8. 9% 


14. 6% 



* S = Student + P Professional 
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TABLE XIX 

CONSISTENCY OF RELEVANCE JUDGMENTS FOR ABSTRACTS 

IN GROUP D (112 ABSTRACTS) 



Relevance Judgment Condition (Frequency %) 
0-0 C-C X-X X-C O-X O-C 



* si 


- S2 


52. 7% 


5.4% 


1. 1% 


3.6% 


26. 8% 


8. 9% 


SI 


- S3 


40. 2% 


7. 1% 


8.0% 


10.7% 


30. 4% 


3. 6% 


si 


- S4 


70.5% 


8.0% 


0. 9% 


8.0% 


10.7% 


1.8% 


si 


- S5 


-54. 5% 


6.3% 


1.8% 


7.1% 


23. .2% 


7. 1% 


si 


- S6 


64. 3% 


7.1% 


5.4% 


8.9% 


9.8% 


4.5% 


S2 


- S3 


40. 2% 


7. 1% 


13. 4% 


7.1% 


23. 2% 


8. 9% 


S2 


- S4 


59. 8% 


6.3% 


0. 0% 


1.8% 


18.8% 


13.4% 


S2 


- S5 


60. 7% 


8.0% 


9.8% 


5.4% 


8. 9% 


7. 1% 


S2 


- S6 


52. 7% 


5.4% 


1.8% 


4.5% 


21.4% 


14.3% 


S3 


- S4 


42. 9% 


12. 5% 


0. 9% 


4. 5% 


35.7% 


3. 6% 


S3 


- S5 


41. 1% 


9.8% 


14.3% 

* 


6.3% 


20. 5% 


8. 0% 


S3 


- S6 


42. 0% 


9.8% 


6.3% 


11.6% 


26. 8% 


3. 6% 


S4 


- S5 


62. 5% 


8.9% 


0. 0% 


3.6% 


15.2% 


9.8% 


S4 


- S6 


70. 5% 


12. 5% 


0. 9% 


3.6% 


8.9% 


3.6% 


S5 


- S6 


52. 7% 


8.0% 


0.9% 


4 . 5% 


21.4% 


12. 5% 


Mean percent 


53. 8% 


8.2% 


4. 5% 


6.1% 


20. 1% 


7.4% 



* S = Student 
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j TABLE XX 

i 

i 

i 

DIFFERENCES OF RELEVANCE JUDGMENTS FOR 

i 

DIFFERENT GROUPS 





Relevance Judgment Condition (Mean %) 
0-0 C-C X-X X-C O-X 


O-C 


Group A 


46. 8% 


10. 9% 


3. 0% 


8. 7% 


17. 9% 


12. 8% 


Group B 


48. 6% 


16. 6% 


2.7% 


6. 1% 


16.7% 


9.3% 


Group C 


47. 7% 


7.7% 


4.7% 


7. 6% 


20. 6% 


11.7% 


Group D 


53. 8% 


8.2% 


4.5% 


6. 1% 


20. 1% 


7.4% 


Composite Mean 


49.2% 


10. 8% 


3.7% 


7. 1% 


18.8% 


10.3% 


Variance 


7.4% 


12.4% 


0.7% 


1.2% 


2.4% 


4.4% 


Standard Deviation 


2. 7% 


3.5% 


0. 9% 


1. 1% 


1.6% 


2. 1% 
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SECTION IV 



CONCLUSIONS 



The results of the experiments on determining document retrievals 
as a function of search strategy and the consistency of relevance judgments 
of information specialists tended to show that overall consistency among 
various individuals is extremely difficult to achieve. Our earlier study on 
indexer consistency showed that the indexing function is probably the easiest 
factor to control. Good indexing consistency is not entirely unexpected, 
since the indexer has available to him the entire text of a document for 
reference in conjunction with the AMIC thesaurus and other reference ma- 
terials. Consequently the meaning and intent of the document usually can 
be recognized and AMIC index terms can be selected with relative ease. 

In the case of retrieving documents from the AMIC system, one is 
provided with a fairly short statement of a request, although at times the 
request statement can be so detailed that improper retrieval terms may be 
suggested. Thus the problem of properly interpreting a search request and 
properly correlating it with the available retrieval terms even with the aid 
of a carefully constructed thesaurus is much more difficult than the indexing 
problem. Perhaps an encouraging result was that even with fairly wide 
variations in strategy, reasonably consistent document returns were obtained. 
The number of relevant documents recalled seems to increase with an in- 
creasing number of individual retrieval terms. This result is not surprising, 
since the possibilities for retrieval increase with an increased number of 
retrieval terms. However, there is the risk of poorer precision. 

The consistency of relevance judgments among information specialists 
is also based on the interpretation of the search request. Furthermore, the 
information specialist also must correlate his relevance judgment based on 
an abstract of the document with his interpretation of the search request. 

Thus the judgment of relevance is probably the most difficult of the three 
tasks of indexing, search strategy formulation and screening for relevance. 
There did seem to be better agreement on judging abstracts to be 
nonrelevant than on judging the relevance of abstracts. 

The experiments pointed out that document retrieval even from the 
well-established AMIC system is a difficult and highly interactive process 
involving the search requester, the information specialist and the information 
storage and retrieval system itself. Because of the many possibilities for 
differences in human expressions, interpretations and judgments, variability 
seems inherent in the overall process of information storage and retrieval. 
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In general it is important that the best possible communication take 
place between the information requester, the information specialist and the 
retrieval system itself. Since the possibilities for variability are recognized, 
the best policy for the information specialist is not to be too restrictive either 
in search strategy formulation or in screening for relevance. The current 
AMIC procedures of using the cutoff feature in searching to achieve maximum 
recall and in providing both relevant and partially relevant abstracts to the 
requester appear to represent an appropriate means of providing the best 
results to the ultimate requester. 
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SECTION V 



DOCUMENT RETRIEVAL SYSTEM OPERATION 



1. INPUT 

During the period covered by this report, 1 Dec 70 through 30 Nov 71, 
8913 documents were indexed and processed into the system. Of this number, 

88 were handbooks, 109 were state of the art, 76 were bibliographies and 
175 were symposia proceedings. The documents were indexed with an average 
of 20. 9 terms per document (exclusive of automatic generic postings) with an 
average indexing time of 31.5 minutes. There are now approximately 60, 000 
documents in the AMIC document retrieval system. 

2. SEARCHING 

A total of 322 technical requests were processed by the Information 
Systems Section during the report period. An average of 9 abstracts was 
printed per search for forwarding to the search requesters. Figure 3 
presents the number of searches processed in each contract year sir.ce 1966. 

3. THESAURUS DEVELOPMENT 

A thesaurus updating was made during the reporting period. New 
terms were added as shown to be necessary from indexing and searching, 
and certain infrequently used terms were deleted from the system after a 
search to determine recent use. Custom thesauri were provided according 
to the preference of individual indexer: 1. separated into three sections - 
general terriiinology, metallic materials terminology and organic chemical 
fragments; 2. two sections with general terminology and chemical fragments 
combined, and the metallic materials terminology printed separately; 

3. only the chemical fragments printed separately; 4. an alphabetical listing 
of the complete thesaurus. A Master Word List was also provided. 

Non-technical special type document retrieval terms are now indicated, 
such as: = = = Bibliography 70 = = = 

Dates are designated as follows: DECADE 1920 - 1929; YEAR I960 

Thirty foreign country terms representing the foreign technology 
source are listed in the introduction to the thesaurus, for example AFRICA, 
AUSTRALIA etc. General non-technical terms such as LITERATURE REVIEW 
are included. Generating or sponsoring agencies that can be recalled speci- 
fically include AEC (Atomic Energy Commission), AFML IN-HOUSE, etc. 

More explanatory notes as well as reference terms are included. The thesaurus 
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provides a high degree of specificity as demonstrated by the following: 

1. Twenty-two military types of aircraft can be searched for: B-52, C5A, 
F-lll, etc. 2. Seventeen coefficient terms including COEF. OF FRICTION, 
DRAG COEF . , HALL COEF, There are 44 listings of collection terms 
ranging from Additives through Waves . Conversion tables are available 
for Energy terms, Mach number, Pressure, Temperature, Frequency 
and Wavelength. One hundred forty-one items under Polymers list a 
specific type or individual trade name as listed in Appendix VI. 
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Figure 3. Documents Indexed and Searches Processed by Year 
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DEFINITION OF TASK NUMBERS 

01 General Includes: 

Supervision 
Meeting & trips 
Holidays & sick leave 
Writing of reports 
Training of students 
Time spent with visitors 

Includes: 

Assignment of accession numbers 
Document accounting records 
Preparation of index and abstract cards 
Indexing 
Keypunching 

Includes: 

Preparation of search strategy 
Search 

Screening of searches 
Search accounting records 
Library loan functions 
Keypunching 

04 Updating Includes: 

Review of vocabulary and thesaurus 
Changes or additions to previous records 
Keypunching 

Acquisition of missing documents 



02 Input 



03 Output 
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05 (UD) Research 



Includes: 



I 

i 

i 

i 



Evaluation studies 
Studies of new techniques 
Investigation of new systems 

05 (AFML/MDC) Includes: 



Preparation of Materials Information Bulletin 

06 Special Projects Includes: 

Work performed in support of the AFML 
not directly related to AMIC retrieval system 

07 Microfilming Includes: 



Time spent on the microfilming of index/ 
abstract records 

08 SDI Includes: 

Preparation of SDI profiles 
SDI records 
Keypunching 

Photocopying of abstracts 
Distribution of abstracts 

16 1498s Includes: 

Keypunching of MASIS Data 
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AMIC 

01 



02 



03 



APPENDIX III 
SUBJECT CATEGORIES 



COSATI 

01 



03, 04 



06, 07 



CATEGORY 



Aeronautics 

Aerodynamics 
Aeronautics 
Aircraft 
Jet engines 

Aircraft flight control and 
instrumentation 

Astronomy, Astrophysics, Atmospheric 
Sciences, Geo-sciences 

Astronomy 

Astrophysics 

Atmospheric physics 

Meteorology 

Geo-sciences 

Chemistry, Biology, Medicine 
Biochemistry 
Bioengineering 
Biology 

Chemical analysis 
Chemical engineering 
Inorganic chemistry 
Life support systems 
Medical science 
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Organic chemistry 



04 



09 



J 05 11A 
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1 IB 



Physical chemistry 

Radiochemistry 

Toxicology 

Electronics, Electrical Engineering 
Components 
Computers 
Devices 

Electrical engineering 

Electronic engineering 

Telemetry 

Adhesives 

Ceramic cements 

Organic resin adhesives 

Potting compounds 

Seals, Sealants 

Ceramic-metal bonds 

Mechanical seals 

O- rings 

Seals, sealants 

Self-healing membranes 

Ceramics, Graphites, Refractories, 
Glasses, Minerals 

Borides 

Carbides 

Carbon, graphites 
Cermets 
49 Glasses 
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Minerals 
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08 



09 



10 

11 



12 



11C 



1 ID 



he 

1 IF 



11H 




Mixed oxides 
Nitrides 
Single oxides 

Coatings, Paints, Oxide Films 
Ceramic coatings 
Cladding 
Coating 

Elastomer coatings 
Oxide films, coatings 
Paints 

Plastic coatings 
Protective coatings 
Reflective coatings 
Composite Materials 
Composites 
Honeycomb 
Laminates 
Sandwich structure 
Fibers, Textiles, Cloth 
Metallurgy, Metallography 
Alloys 
Metals 

Welding, brazing 
Lubricants, Oils 
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Antiseize compounds 



Greases 
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Heat transfer fluids 
Hydraulic fluids 
Lubricants 
Oils 



13 


111 


Polymers , Plastics 


14 


11J 


Elastomers, Rubber 


15 


1 IK 


Cleaning Compounds, Surface Active Agents 


16 


11L 


Wood and Paper Products 


17 


21 


Fuels , Propellants 
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Fuel, aviation 

Fuel, nuclear use category 20 

Fuels 

Explosives 

Liquid propellants 

Solid propellants 

Mechanical, Industrial, Civil, and 
Marine Engineering 

Civil engineering 

Construction equipment, supplies 

Containers, packaging 

Couplings, fittings, joints, 
fasteners 

Industrial processes 

Machining, tools 

Machine elements e.g. bearings, 
gears, gas bearings, etc. 
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Marine engineering 
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Pumps, filters, pipes, tubing 
valves 

Safety engineering 
Structural engineering 
Methods and Equipment 
Apparatus 
Detectors 
Equipment 

Laboratories, test facilities 
and test equipment 

Recording devices 

Testing methods 

Nuclear Science and Technology 

Fuel elements, fuel, nuclear 

Nuclear explosions 

Nuclear power plants 

Nuclear reactors 

Radiation shielding 

Radioactive wastes 

Physics 

Acoustics 

Crystallography 

Electricity and magnetism 

Fluid mechanics 

Lasers and masers 

Optics 
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Particle accelerators 
Particle physics 
Plasma physics 
Quantum theory 
Solid mechanics 
Solid state physics 
Thermodynamics 
Wave propagation 
Space Technology, Missiles 
Astronautics 

Energy conversion, solar cells 
Launch vehicles 
Missile technology 
Re-entry, re-entry vehicles 
Rockets 

Satellites, artificial 

Spacecraft 

Trajectories 
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APPENDIX IV 





SDI SEARCH REQUESTS PROCESSED 
1 DECEMBER 1970 - 30 NOVEMBER 1971 


SEARCH 

NO. 


SEARCH TITLE 


99001 


Polymer Degradation Thermal Stability 


99002 


Vapor Pressure Osmometry 


99003 

99004 


Physics of Polymer Solutions 
Nitroso Copolymers 


99005 


Ladder, Spiro, Thermal Stable Polymers 


99006 

99007 


Properties of Rigid Polymers 
Organic Fluorine Compounds 


99008 

99009 


Ferrocene Compounds Metallocene Polymers 
Spiropolymers Sirocompounds 


99010 

99011 


Ablation, Heat Shields, Thermal Insulation 
Rocket Nozzles, Insulation 


99012 


Advanced Composite Applications 


99013 


Ceramic Reinforcement Materials 


99014 


Properties of Ceramics 


99015 


Hydraulic Fluids, Lubricants 


99016 

99017 


Heat Transfer Fluids, Antiseize Gyro Fluids 
Lubricant Fluid Additives 


99018 

99019 


Lubricant Property Effect on Design 
Hydraulic Fluids and Lubricants 


99020 


Lubricant Behavior Environment 


99021 


Theoretical Analysis of Lubricant Behavior 


99022 


Damping Flotation Fluids 


99023 


Thermal Stability of Polymers 


99024 


Elastomers, Sealants, Polymers 


99025 


Structural Composites 


99026 


Adhesives Surface Preparation 



SEARCH 

NO. 

99027 

99028 

99029 

99030 

99031 

99032 

99033 

99034 

99035 

99036 

99037 

99038 

99039 

99040 

99041 

99042 

99043 

99044 

99045 

99046 

99047 

99048 

99049 

99050 

99051 

99052 

99053 

99054 

99055 

99056 



SEARCH TITLE 
Elas tomers, Sealants, Rubber 
Engineering Properties of Metal Alloys 
Structural Composites 
Composites for Rocket Insulation 

Properties of High Temperature Polymer Composites 

Thermal Decomposition of Polymer Composites 

Testing of. Polymer Composites 

Processing Polymer Composites 

Glass Technology, Properties 

Transparent Materials 

Transparent Films for Windows 

Attachment Design of A/C Windows 

Glass Technology, Properties 

In-Service Failure Analysis 

Cleaning of Aircraft 

Stress Corrosion In-Service 

Measurement of Surface Properties 

Metal, Polymer Composites 

Structural Adhesives 

Transparent Materials 

Armor 

Radar Absorbing Materials 
Radomes 

Composites Reinforcement Interfaces 
Elastomeric Materials 

Flame Resistant Liquid Propellant Compatible Elastomers 
Tires 

Viscoelastic Damping Materials 
Fluid Seals and Sealants 
Block and Graft Polymers 
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SEARCH 

NO. 

99057 

99058 

99059 

99060 
99061 
99062 

99063 

99064 

99065 

99066 

99067 

99068 

99069 

99070 

99071 

99072 

99073 

99074 

99075 

99076 

99077 

99078 

99079 

99080 

99081 

99082 

99083 

99084 

99085 

99086 



SEARCH TITLE 

Fuel Cells 

Expulsion Bladders 

Electrical Insulation Material 

Ablative Elastomeric Rocket 

Rain Erosion Resistant Materials 

Mechanical Properties Evaluation of Metals 

Mechanical Properties Evaluation of Composites 

Compatibility of Metals Composites With Oxidizers 
Propellants 

Structural Application of Metals Composites Aerospace 
Vehicles 

Failure Analysis of Metals Composites 
Fabrication of Metals Composites 
Hypervelocity Impact 
Shock Impulsive Loading Phenomena 
Aircraft Armor Materials Impact 
Carbon Fiber Research Technology 
Three Dimensional Fabrics 
Design Properties Composites 
Ablation Thermal Degradation 
Fibers 

Structural Polymer Composites 
Micromechanics, Mechanics Composites 
Mechanics, Micromechanics Composites 
Photosensitive Compounds 
Carbon Fibers Pyrolysis of Organic Fibers 
Transpiration Cooling 

Environmental Effect on Fibrous Materials 
Fabric Properties 
Flammability of Materials, Fabrics 
Recovery, Safety of Personnel 
Parachute System Loading 



SEARCH 

NO. 

99087 

99088 

99089 

99090 

99091 

99092 

99093 

99094 

99095 

99096 

99097 

99098 

99099 

99100 

99101 

99102 

99103 

99104 

99105 

99106 

99107 

99108 

99109 

99110 

99111 

99112 

99113 

99114 

99115 

99116 



SEARCH TITLE 

Expandable Structures 
Coated Fabrics 

Properties High Strain Rate Fibers 
Ceramics, Polymers, Nonmetallics 
Liquid Crystals, Photochemistry 
Inorganic Nonmetallic Reinforcing Fibers 
Polymeric High Strength Fibers 
Fiber Optics 

Electrically Conductive Fibers 

Properties of Ceramics 

Intermetallic Compounds 

Plastic Radomes for Aircraft Missiles 

Compressor Blades for Aircraft Engines 

J oining, Welding, Brazing 

Stress Corrosion Cracking 

Shells; Panels, Structural 

Shock Phenomena 

Vibration Damping 

Mechanical, Physical Properties Testing 
Dynamic Stress Properties 
Reliability 

Mathematical Model Life Prediction 
Heat Resistant Polymers 
Thermoplastics 

Polymer Composites Not B Fibers 
Forming of Metals 

Microstructure Crystal Structure Metals 
Metal Matrix Composites 
Strengthening Metallurgical Reactions 
Metallurgy of Various Metals Alloys 
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SEARCH 



NO. 


SEARCH TITLE 


99117 


Powder Metallurgy Techniques 


99118 


Diffusion Data On Metals Alloys 


99119 


Hydrogen Embrittlement Stress Corrosion 


99120 


Environmentally Induced Failure 


99121 


Mechanical Properties of Metals Alloys 


99122 


Thermomechanical Processing 


99123 


Thermodynamics of Borides 


99124 


Coating Processes 


99125 


Corrosion Mechanism and Protection 


99126 


High Temperature Plastic Coatings 


99127 


Thin Metal Films 


99128 


Paints, Primers, Surface Finish 


99129 


Paint Formulation Ingredients 


99130 


Thermal Controls, Coatings 


99131 


Rain Resistant Electrical Dissipating Coatings 


99132 


Ablation, Composites, Re- Entry 


99133 


Coatings, Erosion,Infrared Gun 


99134 


Polymer Composite Tankage 


99135 


Batteries Materials 


99136 


Nondestructive Testing Inspection 


99137 


Energy Interactions With Materials 


99138 


Emission From Stressed Materials 


99139 


Production Quality Control Materials 


99140 


Ceramics, Ceramic Materials 


99141 


C e rmets, Inter metalli c s 


99142 


Ceramic Coatings 


99143 


Flow, Wear, Fracture Behavior Ceramics 


99144 


Superconductor, Semiconductors, Ceramics 


99145 


Ceramic Composites, Reinforcement 


99146 


Characterization, Chemical Behavior Ceramics 


99147 


Instrumentation for IR Spectra 


99148 


Molecular Crystal Structure Vibration 
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SEARCH 

NO. 

99149 

99150 

99151 

99152 

99153 

99154 

99155 

99156 

99157 

99158 

99159 

99160 

99161 

99162 

99163 

99164 

99165 

99166 

99167 

99168 

99169 

99170 

99171 

99172 

99173 

99174 

99175 

99176 

99177 

99178 

99179 



SEARCH TITLE 

Infrared Spectra, Spectrometry 
Microwave, Electronic Development and Material 
Electrical Power Devices, Electro-Chemical 
Radomes High Temperature Dielectrics 
Lasers, Materials and Effects 

Photoconductivity Silicone Semiconductors Delay Lines 

Semiconductor Materials Properties 

Magnetism Magnetic Materials 

Intermetallic 3D-4F Compounds 

Intermetallic Magnetic Materials 

Magnets, Magnetic Properties 

Semiconductors, Related Phenomena 

Magnets Rare Earth Intermetallic s 

Semiconductors Related Phenomena 

Boundary Layer Flowj Vehicles, Hypersonic 

Infrared Thermal Optics of Bodies 

Ablation, Thermal Protection 

Thermal Protection Systems 

Fluoro Organic Compounds 

Fluorinated Polymers 

Melting of Metals And Alloys 

Metal Working Processes 

Lubrication for Metal Working 

Thermo-mechanical Processing of Metals 

Material Properties at Processing Condition 

Defects Arising from Metal Processing 

Mathematical Analysis of Metal Working 

Ablation of Composites, Graphites 

Temperature Measuring Instrumentaion 

Carbon Fibers Chemical Analysis Decomposition from PAN 

Polyacrylonitrile Decomposition 



SEARCH 



NO. 


SEARCH TITLE 


99180 


Rain^Dust Erosion Phenomena 


99181 


Dynamic Response Properties of Metals 


99182 


High Temperature Ceramics 


99183 


CeramiC|Metal Composites 


99184 


Orthopedic Implant Materials 


99185 


IR Sign Optics Contamination 


99186 


Mechanical Properties of Graphites, Composites 


99187 


Thermal Stress } Stress Analysis 


99188 


Ablation of Composites, Graphites 


99189 


Thermophysical Properties 


99190 


Hypervelocity Impact 


99191 


Ablation, Heat Transfer, Re-Entry 


99192 


Powder Metallurgy Technology 


99193 


Compatibility of Metals, Composites with Oxidizers, 
Propellants 


99194 


Structural Application of Metals, Composites Aerospace 
Vehicles 


99195 


Failure Analysis of Metals Composites 


99196 


Fabrication of Metals Composites 


99197 


Crystal C Fibers Thermal Analysis 


99198 


Rare Earth Alloys Crystal Structure 


99199 


Rare Earth Co Magnetic Materials 


99200 


Holography, Crystal Deformation 


99201 


Ceramic Coatings Flame Spray 


99202 


Mechanical Properties of Mg O Glasses 


99203 


Design of Instrumentation 


99204 


Aerothermodynamics 


99205 


Liquid Vapor Phase Transitions 


99206 


Optical Properties 


99207 


Fiber Optics 


99208 


Solar Radiation Solar Furnace 


99209 


Nuclear Aircraft Antenna Harding 



60 

;v 66 

v v r 



SEARCH 

NO. 

99210 

99211 
99212 

99213 

99214 

99215 

99216 

99217 

99218 

99219 

99220 
99221 
99222 

99223 

99224 

99225 

99226 

99227 

99228 

99229 

99230 

99231 

99232 

99233 

99234 

99235 

99236 

99237 

99238 



SEARCH TITLE 

High Temperature Oxidation Kinetics 
Absorption Cross Section IR,UV 
Cryogenics, Cyrogenic Insulation 
Mass Spectrometry Mossbauer 
Temperature Measurement 
Thermal Control Coatings 
Thin Films 

Electronic Countermeasures 

Gas Chromatography, Decomposition of Polymers 

Ablation Transpiration Cooling 

Re-Entry Ballistic and Lifting 

Hypersonic Vehicle Boundary Layers 

Ablation, Especially High Pressure Ablation 

High Temperature Thermal Insulation 

Thermal Optical Properties - IR 

Ceramics, Graphites for Thermal Protection 

Nondestructive Biaxial Testing 

Mechanical Properties of Graphite C/C Materials 

Thermophysical Properties 

Thermal Stress, Stress Analysis 

Thermionic Tubes, Mate rials and Processes 

Fabrication Process, Electronic Components 

Physical Properties of Magnetic Materials 

Ceramic Substrates Packaging for Magnetic Devices 

Dielectric for Magnetic Devices 

Garnets and Ferrites 

Semiconductor Compounds 

Energy Conservation Materials 

Masers and Lasers 



SEARCH 




NO. 


SEARCH TITLE 


99239 


Luminesence, Optic Property Special Material 


99240 


Glass es-Fabrication, Properties 


99241 


Design Behavior of New Composites 


99242 


Advanced Structural Composites 


99243 


Metal Processing 


99244 


Photosensitive Materials 


99245 


Elastomeric Fluid Seals 


99246 


NMR, EPR Spectra and Phenomena 


99247 


Polymeric Dielectric Coatings 


99248 


Elastomers and Applications 


99249 


Temperature Control Coatings 


99250 


Paint Coating Formulation Camouflage 


99251 


Thin Metal Foils Preparation 


99252 


High Temperature Corrosion Protective Coatings 


99253 


PhysicSjMolecular Weight Determination - Polymer Solutions 


99254 


Molecular Vibration Spectra of Materials 


99255 


Instrumentation for IR Spectra 


99256 


Ablation, Re-Entry Phenomena 


99257 


Nonmetallic Radomes Fabrication 


99258 


IR Ero% Oxidation Resistant Coatings 


99259 


Decomposition of Polymers 


99260 


Protection Materials for Rocket Nozzles 


99261 


Emission From Materi als Under Stress 


99262 


Nondestructive Testing,Quality Control 


99263 


Energy Effects on Materials 


99264 


Reinforced Metal Polymer Composites 


99265 


Ablation,High Temperature Behavior 


99266 


Metallurgy, Behavior of Metals 


99267 


Diffus ion, Str e s s C o r r o sion, Embrittlement 


99268 


Metal Matrix Composites 
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SEARCH 

NO. 


SEARCH TITLE 


99269 

99270 

99271 

99272 

99273 

99274 

99275 

99276 

99277 

99278 

99279 

99280 

99281 
99282 

99283 

99284 


Vibration, DampingjNoi.se 

Mechanical Properties Testing 

Dynamic Load Behavior of Materials 

Math, Statistical Prediction of Behavior 

Ablation Re-Entry Nose Tips 

Ceramic, Graphite Materials 

Ceramic Composites Armor 

Purity Nitroso Elastomers Benzidine 

Manufacturing Technology of Propellants 

Manufacturing Technology of Nonmetallic Materials 

Ceramic Materials Properties Application 

Cratering 

Plastic Cartridge Cases 
Shock Dynamic Loading of Materials 
Hypervelocity Impact 
Lubricants and Related Materials 


99285 

99286 

99287 

99288 

99289 

99290 


Theoretical Study of Lubricants 

Ceramics Properties and Applications 

Joining, Welding of Metals 

Metal Composites 

Carbides, Cermets Phase Diagram 

E-M Window Material for IR Laser Radiation 


99291 

99292 

99293 

99294 

99295 

99296 


Optical Properties of Inorganic Materials 
Solar Radiation Energy 

High Temperature Oxidation Reaction Kinetics 
Radar IR UV Absorption Materials 
Cryogenic High Temperature Thermal Insulation 
Time of Plight Mass Spectrometry 



SEARCH 

NO. 

99297 

99298 

99299 

99300 

99301 

99302 

99303 

99304 

99305 

99306 



SEARCH TITLE 
Noise Pollution 

Chemical, Physical Behavior in Ablative Wakes 

Fibrous Materials for Clothing 

Lasers, Laser Window Materials 

Processing of Ablative Composites 

Ablation Phenomena, Mechanism 

Adhesives, Properties and Interfacial Phenomena 

Properties of Structural Polymer Composition 

Effect of Electrical Field on Interfaces 

Composites 
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APPENDIX V 

RETROSPECTIVE SEARCH REQUESTS PROCESSED* 
1 DECEMBER 1970 - 30 NOVEMBER 1971 



SEARCH 

NO. 


SEARCH TITLE 


2218 


Asbestos Reinforced Plastics 


2219 


Diffusion Bonding Aluminum 


2220 


Properties and Technique of Carbon 


2221 


Foaming of Liquids 


2222 


Niobium Phase Diagram 


2223 


Oxidation of Niobium 


2224 


Thermal Conduction Plastics 


2225 


Fatigue of Aluminum 7075-T76 


2226 


Protective Coatings for Styrofoam 


2227 


Hypervelocity Flow by Electrofluid Dynamics 


2228 


Computerized Analysis 


2229 


Shot Peening Landing Gear 


2230 


Heat Treatment Effect on Ti-6-4 


2231 


Molding Miniature Gears 


2232 


Ceramics in Jet Engines Turbine Blades 


2233 


Electrical Discharge Machining 


2234 


Fatigue Testing of Polymer Composites 


2235 


Glass -Ceramic Enclosures 


2236 


Physiological Effects of Ozone + NO 


2237 


Zinc Chromate Removers 


2238 


Sources and Measurement of Ozone + NO 


2239 


Foreign Technology 


2240 


Ultrasonic Soldering 


2241 


Meteorid Impacts 


2242 


Vapor Thermal Conductivity 


2243 


Rare Earth Glasses 


2244 


Energy Crisis Alternatives 


2245 


Solar and Geyser Power 


2246 

* 


Gelcel Battery Information 



Includes special searches run by UD for updating and for research 
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SEARCH 

NO. 

2247 

2248 

2249 

2250 

2251 

2252 

2253 

2254 

2255 

2256 

2257 

2258 

2259 

2260 
2261 
2262 

2263 

2264 

2265 

2266 

2267 

2268 

2269 

2270 

2271 

2272 

2273 

2274 

2275 

2276 

2277 



SEARCH TITLE 
Wear Resistant Coatings 
IR Detectors, Lasers, Emissivity 
Injected Molding Fiberglas 
Polyvinyl Chloride Pipes 
Audio Frequency 20-20K 
Lightweight Shelter Materials 
Fatigue Cu Alloys 
Thermal Fatigue Cu Alloys 
Welding Steels and Ti Alloys 

Standards for High Pressure Pumps Fluid Motors 
Ballistics 

Fatigue Cu-Ag-Zr Alloys 
Thermal Fatigue Cu Metal Systems 
Liquid Crystals 

Foreign Technology-Update Term 
Supercooling - Update 
Work Functions - Update 
Superheating - Update 
Polysulfones - Update 
Polarons - Update 
Maneuverability 
Arsenites - Update 
Chemical Reactivity - Update 
Fiber Reinforced Composites 
Aluminum Phosphate Properties 
Jet Blast Deflection Coatings 
Properties of CdTe,ZnSe 
Non-Aqueous Electrochemical Cells 
Beryllium Nb - Update 
Aircraft B1A - Update 
Work Hardening Coefficient - Update 
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SEARCH 

NO. 



SEARCH TITLE 




2278 

2279 

2280 
2281 
2282 

2283 

2284 

2285 

2286 

2287 

2288 

2289 

2290 

2291 

2292 

2293 

2294 

2295 

2296 

2297 

2298 

2299 

2300 

2301 

2302 

2303 

2304 

2305 

2306 

2307 

2308 



Chemical Reactors - Update 
Fatigue Properties of Steel 8620 
Fracture Behavior of Composite 
Hydrogen Phosphates - Update 
Hydrogen Carbonates - Update 
Hydrogen Arsenates - Update 
Hydrogen Sulfate - Update 
Honeycomb - Standards, Quality Control 
Bor on, Graphite Fiber Fabricate 
Void Content In Composites 
Costs of Filaments 
Environmental Effects 
Impact Effects on Composites 
TD Nickel, TD Nickel Chromium 
Dispersion Hardening Theory 
Nichrome, Inconel Mechanical Properties 
Catalysis of Graphitization 
Self-Sealing Fuel Lir.es 
High Intensity Radiation Effects 
Electrically Conductive Plastics 
Transparent Plastics 
Adhesive Bonded Components 
High Performance Thermoplastics 
Polyester - Wool Fibers 
Properties of Carbon Fibers 
Fiber Formation 
Degradation of Plastics 
Permeability of Coatings 
Stress Corrosion Coating 

Metallic Corrosion Due to Polymer Composites 
Reliability Theory of Random Fatigue 
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SEARCH 

NO. 

2309 

2310 

2311 

2312 

2313 

2314 

2315 

2316 

2317 

2318 

2319 

2320 

2321 

2322 

2323 

2324 

2325 

2326 

2327 

2328 

2329 

2330 

2331 

2332 

2333 

2334 

2335 

2336 

2337 

2338 

2339 

2340 

2341 



SEARCH TITLE 
Fiber Reinforced Plastic Pipe 
Intermetallic Alloys 
PAN Oxidation, Degradation 
Electrical Resistant Carbon Graphite Fibers 
Coating, Aircraft vs Nuclear Flash 
Nuclear Flash Curtain 
UV Polymer Film in Vacuum 
Flammability of Textiles 
Static Electric Buildup 
Thermal Radiation Sources 
Ballistic Impact 
Low Cycle Fatigue Copper-Zr 
Thermal Fatigue Copper-Zr 
Nitinol 55 

Thermal Fatigue^ Copper-Ag-Zr 
Thermal Fatigue, Dispersion Hardening Cu 
Fatigue Copper-Ag-Zr 

Fatigue, Dispersion Hardening of Copper Alloys 
Rare Earth Magnets 

Refractory Materials as Transpiration Matrices 

Transpiration Film Cooling 

Optical Properties IR Glasses 

Characterization of Solids 

Superconducting Materials 

Cobalt-Rae Magnets 

Cathode Materials - Batteries 

Carbon Insulation - Heat Shields 

Composite Armor 

Windows for High Power Lasers 

Electrical Insulation 

Potting Compounds for Aircraft 

Growth Technique for Telluride 

Ultrasonic Nebulization 
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SEARCH 

NO. 


SEARCH TITLE 




2342 


Thin Film Beryllides, Carbides 




2343 


Electric Discharge Initiated Reactions 




2344 


Electric Discharge Initiated Reactions 




2345 


Tungsten Thermal Conductivity, Electrical Resistivity 




2346 


Oxidation of Ceramic Composites 




2347 


Plastic Armor Backing 




2348 


Integral Armor 




2349 


Reflective Materials Barium as Electromagnetic Radiation 




2350 


Quartz Fiber Quartz Matrix AS3DX 




2351 


Hypersonic Insulation Material 




2352 


Solidification of Eutectics 




2353 


Laser Spectroscopy - Solutions 




2354 


Properties of Polymers 




2355 


Fatigue Failure Low Alloy Steels 




2356 


Stainless Steel 304 Fatigue 




2357 


Properties of Astralloy 




2358 


B-Phase Vandium - 3Al 




2359 


Adhesive Bonded Joints 




2360 


Dielectric Properties Adhesive Bond 


: 


2361 


Fatigue Adhesive Bonded Joints 




2362 


Thermal Properties of Adhesive 




2363 


Impact Strength of Adhesives 




2364 


Heat Resistant Plastics 


i 

t 


2365 


Additives for Adhesives 


j 


2366 


In-Space Adhesives 


1 

t 


2367 


Surface Treatment for Metals 


r 


2368 


High Temperature Adhesives 


i 

r. 


2369 


Organosilicon Polymers 


r 

u 


2370 


Cellular Strucutral Materials 


y 

i 


2371 


Intumescent Coatings 


{ 

i 


2372 


Inflatable Impact 


t 


2373 


Properties Textile Materials 


V 

l 

t 


2374 


Textile Coating Properties 
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SEARCH 

NO. 


SEARCH TITLE 


2375 


Heat Transfer in Textiles 


2376 


Textiles 


2377 


Thermal Protection Textiles 


2378 


1, 3 Addition Reactions 


2379 


Photochemical Ring Closures 


2380 


Diels - Alder Reactions 


2381 


Room Temperature Curing Epon 


2382 


Toxic Fumes in Composites 


2383 


Bolts 


2384 


D e contam ination 


2385 


UV or IR Pumped Phosphors 


2386 


Polishing Zinc Telluride 


2387 


Fracture in Ti,Al, Mo, V, Alloys 


2388 


Scanning Electron Microscopy 


2389 


Infrared Nondestructive Tests 


2390 


Titanium, Titanium - Mg 


2391 


Titanium - Mg Part II 


2392 


Pyrolyzed Plastic Composites 


2393 


3-D Reinforcement Composites 


2394 


Aromatic and Heterocyclic Polyme 


2395 


Ablation, Ablative Materials 


2396 


Missile Thermal Protection 


2397 


Analytical Modeling 


2398 


Thermal Protective Materials 


2399 


Thermal Conductivity Fiberglas 


2400 


Toxicity of Aircraft Materials 


2401 


Magnesium Effect on Titanium 


2402 


Carbon Composites 


2403 


Carbon-Carbon Thermal Systems 


2404 


Fluorimetry of Phosphates 


2405 


Pollution 


2406 


Plastic Composite Properties 
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SEARCH 

NO. 


SEARCH TITLE 


2407 


Phase Equilibrium of Rae Oxides 


2408 


Nonmetallic Cutting Tools 


2409 


Cryogenic Liquid Helium Seals 


2410 


Fatigue Stainless 15-7 


2411 


Gyro Bearings 


2412 


Lubricant Composites 


2413 


Instrument Bearings 


2414 


Oxide Reinforcement 


2415 


Reactions of Oxides 


2416 


Task I S. Rusek 


2417 


Ta3k I J. M. Ascenzi 


2418 


Task I J. F. Hoffman 


2419 


Task I C. M. Erdman 


2420 


Task I R. Graves 


2421 


Task I T. E. Moloney 


2422 


Task II R. Graves 


2423 


Task II J. M. Ascenzi 


2424 


Task H J. F. Hoffman 


2425 


Task H S. Rusek 


2426 


Task II T. E. Moloney 


2427 


Task II C. M. Erdman 


2428 


Fuel Permeability 


2429 


Task III R. Graves 


2430 


Task III J. M Ascenzi 


2431 


Task HI J. F. Hoffman 


2432 


Task HI S. Rusek 


2433 


Task HI C. M. Erdman 


2434 


Task HI T. E. Moloney 


2435 


T-M Technique for Alpha-Beta Ti 


2436 


Ammonium Perchlorate 


2437 


Graphite Monofilaments 


2438 
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The ability of various AMIC information specialists to prepare search 
strategies for document retrieval was studied by providing ten typical search re- 
quest statements to seven information specialists. Each specialist prepared search 
strategies independently. Significant variations occurred among the strategies, 
although even with these variations, reasonably consistent document returns result- 
ed. The experiment indicated that the proper interpretation of a search request and 
the conversion of the request to an appropriate search strategy even with a well- 
established thesaurus is a considerably more difficult task than indexing. In another 
experiment, the consistency of relevance judgments among information specialists 
was examined. Relevance judgment is more difficult than indexing and search 
strategy preparation. Agreement on nonrelevance is better than agreement on 
relevance. The system of relevance judging according to nonrelevant, partially 
relevant and relevant appears optimum for AMIC. Communication between the 
requester, the information specialist and the retrieval system itself are of prime 
importance. Because of certain human interactions which must occur, some 
variability is unavoidable. However, current AMIC procedures of searching to 
achieve high recall and screening the results for relevance provide good results to 
the ultimate requester. 
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